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DETAILED ACTION 
Response to Amendment 

Applicant's submission filed on January 16, 2008 has been entered. Claims 1-17, 25 and 
32 have been canceled. Claims 18-21, 26-28, 30 and 33 have been amended. Claims 34-35 have 
been newly added. Claims 18-24, 26-31 and 33-35 are pending in the application. 

Response to Arguments 

Applicant's arguments filed January 16, 2007 have been fully considered but they are not 
found persuasive in view of the new ground(s) of rejection set forth in the present Office Action. 

Applicant argues in essence with respect to the claim 18 and similar claims that, "even 
assuming, arguendo, Feiner and Filo could have been combined in the manner proposed in the 
Office Action, it is submitted that such a combination still fails to teach or suggest, among other 
features, a generation unit that 1) generates a virtual space image to be displayed by the display 
unit according to a first procedure and 2) generates a virtual space image to be output according 
to a second procedure which is different from the first procedure, as set forth in Claim 18 of 
Applicants' invention." The examiner respectfully disagrees with this argument. 

Feiner teaches in Page 79 that the HTTP server can call a network object method that 
selects corresponding graphical items on the head-worn display. Thus data selection works in 
both directions: from the backpack PC to the handheld PC by launching relevant URLs from the 
head- worn display's menus and vice versa by selecting buildings, departments, etc on the head- 
worn display from a link on the handheld's browser. Feiner teaches that the tour application 
continuously receives input from the GPS position tracker and the orientation tracker and 
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therefore, when the instruction is received from said instruction unit (e.g., such as a printing 
instruction in a graphical user interface), the mixed reality image is re- generated using a second 

calculation method a second procedure — and sequence such as the blending the graphics 

image so re-generated with a different building image and different labels are generated to 
overlay on the building image and thus requiring different calculation methods for calculating 
the network objects or generating the graphics images to be overlaid on the building images- — 
different procedures/methods for generating the mixed reality space images. Figs. 2-3 show the 
mixed reality images according to a first viewpoint and a second viewpoint. Based on this input 
and a database of information about campus buildings, it re-generates the graphics that are 
overlaid on the real world by the head-worn display. At Page 75 and Section 3, Feiner discloses a 
browser running on the handheld computer that generates a page of mixed reality on the fly and 
the web browser allows the page be printed on a printer connected to the handheld computer on a 
network of base stations on campus wherein the network inherently has a plurality of printers 
wherein a printing instruction is issued and then the instruction is sent over the network so as to 
allow a mixed reality space image to be printed in a network printer on the campus-wide 
network — the determination of the overlay information in the blending for providing a mixed 
reality space image allows a mixed reality space image to be re-generated differently from the 
previous one being generated according to a first calculation method/procedure. Thus, the second 
procedure for generating another mixed-reality space image is clearly different from the first 
procedure for generating a mixed-reality space image as different calculation methods are 
employed. Moreover, the printing procedure for printing the virtual space image allows the 
generation unit to provide a virtual space image different from the one generated by the 
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generation unit according to the different calculation methods because virtual space images are 
continuously updated and the different calculation methods are employed. See column 2 of Page 
78. Since the superposed image is displayed on a web browser, it can be printed on a printer on 
campus network anywhere and anytime as the virtual space image is continuously being updated. 
Therefore, Feiner teaches the claim limitation of "a second procedure which is different from the 
first procedure". 



Claim Rejections - 35 USC §103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 18-24, 26-31, and 33-35 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Steven Feiner, Blair Maclntre, Tobias Hollerer, and Anthony Webster, 
"Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment", 
Personal Technologies, vol. 1, no. 4, pp. 208-217, 1997 (hereinafter Feiner) in view of Filo et al. 
U.S. Patent No. 6,215,498 (hereinafter Filor). 

Re Claims 18-19 and 26-27 and 33-35: 

Feiner teaches an image processing apparatus which is connected to print unit adapted to 
print an image, and can output an image to be printed to the print unit (See Page 75 at Section 3, 
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wherein Feiner discloses a browser running on the handheld computer that generates a page of 
mixed reality on the fly and the web browser allows the page be printed on a printer connected 
to the handheld computer on a network of base stations on campus wherein the network 
inherently has a plurality of printers; see column 2 of Page 78), comprising: 

An image sensing unit adapted to capture an image of a real space {Page 78, Section 4.1 
including a position and orientation tracker with HMD); 

A generation unit adapted to generate an image of a virtual space according to a 
viewpoint position and orientation of said image sensing unit (Page 78, Section 4.1 including a 
position and orientation tracker with HMD and the graphics image of a virtual space is 
generated in accordance with the position and orientation of the trackers. See Section 4.2, based 
on the user input and a database of information about campus buildings, it generates the 
graphics that are overlaid on the real world by the head-worn display. See Section 4.3 wherein 
the images are generated by the CCD camera. Overlaying virtual objects or graphics and textual 
texts on the real world images shot by the sensing unit on the head-worn display; see Section 3 
wherein the real world image were shot through the head-worn display); 

A composition unit adapted to obtain a mixed reality space image by combining the real 
space image captured by said image sensing unit and the virtual space image generated by said 
generation unit (See Section 4.2, based on the user input and a database of information about 
campus buildings, it generates the graphics that are overlaid on the real world by the head-worn 
display. See Section 4.3 wherein the images are generated by the CCD camera. See Section 3; 
for example, overlaying the textual labels on automatically recognized campus buildings shown 
in Figures 2 and 3 wherein the building is selected based on the viewpoint of the user 's head 
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position and orientation and the mixed reality image is stored at least in a cache memory or the 
frame buffer of the head-worn display and displayed on the head-worn display. As shown in Fig. 
7, the HTTP server has two components, the campus information server, responsible for the 
dynamic generation of HTML pages, and a caching proxy server. The purpose of the proxy 
server is to cache the data returned by external HTTP requests to mitigate the slowness of the 
radio network link. Commonly accessed pages, such as department home pages and building 
descriptions, can be pre-cached without relying on the browser's own caching mechanisms); 

A display unit adapted to display a mixed reality space image obtained by said 
composition unit (At Page 75 and Section 3, Feiner discloses a browser running on the handheld 
computer that generates a page of mixed reality on the fly and the web browser allows the page 
be printed on a printer connected to the handheld computer on a network of base stations on 
campus wherein the network inherently has a plurality of printers; see column 2 of Page 78; On 
the web page, the user instructs whether or not the mixed reality space image is to be printed or 
not and the user interface or the web browser allowing the instruction to be made constitutes the 
instruction unit allowing the user to instruct whether or not the mixed reality space image is to 
be printed or not. Since the superposed image is displayed on a web browser, it can be printed 
on a printer on campus network); 

An output unit adapted to output a mixed reality space image, which is obtained by said 
composition unit, to the print unit in response to an input of a print instruction (At Page 75 and 
Section 3, Feiner discloses a browser running on the handheld computer that generates a page 
of mixed reality on the fly and the web browser allows the page be printed on a printer 
connected to the handheld computer on a network of base stations on campus wherein the 
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network inherently has a plurality of printers ; see column 2 of Page 78; On the web page, the 
user instructs whether or not the mixed reality space image is to be printed or not and the user 
interface or the web browser allowing the instruction to be made constitutes the instruction unit 
allowing the user to instruct whether or not the mixed reality space image is to be printed or not. 
Since the superposed image is displayed on a web browser, it can be printed on a printer on 
campus network); and 

Wherein said generation unit generates a virtual space image to be displayed by said 
display unit according to a first procedure {e.g., Section 4.2 wherein a system that provides 
language-level support for distributed virtual environment and CORTERIE is based on the 
distributed data-object paradigm for distributed shared memory and the tour application 
running on the backpack PC is response for generating the graphics and presenting it on the 
head-worn display and the HTTP server can call a network object method that selects 
corresponding graphical items on the head-worn display and the handheld display controls the 
head-worn display. The tour application continuously receives input from the GPS position 
tracker and the orientation tracker and thus continuously updating the mixed reality image 
composed by the graphics image generated from the handheld computer and the HTTP server 
and the building image shot from the camera; see Figs. 2-3. It also takes user input from the 
trackpad that is physically attached to the back of the handheld PC. Based on this input and a 
database of information about campus buildings, it generates the graphics that are overlaid on 
the real world by the head-worn display. See also Section 3, when a building is selected, a 
conical green compass pointer appears at the bottom of the head-worn display, oriented in the 
building 's direction. The pointer turns red if the building is more than 90 degrees away from the 
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user 's head orientation. This allows the user to find the building more easily if they turn away 
from it. The pointer is especially useful for finding buildings selected from the hand-held 
computer. This is made possible by our custom HTTP server, which can tell the backpack 
computer to select a building on the head-worn display. See Section 4.2, the replicated shared 
objects support asynchronous data propagation with atomic serializable updates and 
asynchronous notification of updates), and said generation unit generates a virtual space image to 
be outputted by said output unit according to a second procedure which is different from the first 
procedure {At Page 75 and Section 3, wherein Feiner discloses a browser running on the 
handheld computer that generates a page of mixed reality on the fly and the web browser allows 
the page be printed on a printer connected to the handheld computer on a network of base 
stations on campus wherein the network inherently has a plurality of printers; clearly the 
procedure employed in generating the virtual space image to be printed on an output unit is 
different from the generation procedure for generating another virtual space image as different 
calculation methods are employed as the virtual space images are continuously updated. See 
column 2 of Page 78; Since the superposed image is displayed on a web browser, it can be 
printed on a printer on campus network). 

Feiner teaches in Page 79 that the HTTP server can call a network object method that 
selects corresponding graphical items on the head-worn display. Thus data selection works in 
both directions: from the backpack PC to the handheld PC by launching relevant URLs from the 
head- worn display's menus and vice versa by selecting buildings, departments, etc on the head- 
worn display from a link on the handheld's browser. Feiner teaches that the tour application 
continuously receives input from the GPS position tracker and the orientation tracker and 
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therefore, when the instruction is received from said instruction unit (e.g., in a graphical user 

interface), the mixed reality image is re-generated using a second calculation method a 

second procedure — and sequence such as the blending the graphics image so generated with a 
different building image and different labels are generated to overlay on the building image and 
thus requiring different calculation methods for calculating the network objects or generating the 
graphics images to be overlaid on the building images — different procedures/methods for 
generating the mixed reality space images. Figs. 2-3 show the mixed reality images according to 
a first viewpoint and a second viewpoint. Based on this input and a database of information 
about campus buildings, it generates the graphics that arc overlaid on the real world by the head- 
worn display. At Page 75 and Section 3, Feiner discloses a browser running on the handheld 
computer that generates a page of mixed reality on the fly and the web browser allows the page 
be printed on a printer connected to the handheld computer on a network of base stations on 
campus wherein the network inherently has a plurality of printers wherein a printing instruction 
is issued and then the instruction is sent over the network so as to allow a mixed reality space 
image to be printed in a network printer on the campus-wide network - — the determination of the 
overlay information in the blending for providing a mixed reality space image allows a mixed 
reality space image to be generated differently from the previous one being generated according 
to a first calculation method/procedure. Thus, the second procedure for generating another 
mixed-reality space image is clearly different from the first procedure for generating a mixed- 
reality space image as different calculation methods are employed. Moreover, the printing 
procedure for printing the virtual space image allows the generation unit to provide a virtual 
space image different from the one generated by the generation unit according to the different 
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calculation methods because virtual space images are continuously updated and the different 
calculation methods are employed. See column 2 of Page 78. Since the superposed image is 
displayed on a web browser, it can be printed on a printer on campus network anywhere and 
anytime as the virtual space image is continuously being updated. Therefore, Feiner teaches the 
claim limitation of "a second procedure which is different from the first procedure". 

Although Feiner does not explicitly disclose "the print unit", Feiner's apparatus is 
connected to the campus network and the campus network inherently has a plurality of printers 
connected to allowing Feiner's web browser to print the mixed reality image upon user's 
instruction. Nevertheless, Filo discloses the print unit (See Filo column 11, lines 10-26). Filo 
teaches other claim limitations set forth in the claim 1 and similar claims as well by disclosing 
the superposition of the virtual objects with the real objects. 

It would have been obvious to one of the ordinary skill in the art to have used a printer to 
print the mixed reality image for review at a later time using the web browser and a network 
printer (See Feiner Page 78 and Filo column 11, lines 10-26). One of the ordinary skill in the art 
knows how to connect a printer to a handheld computer. One of the ordinary skill in the art 
would have been motivated to do so to print the mixed reality image, e.g., in the interest of 
showing the campus building to a third party (See Feiner Page 78 and Filo column 11, lines 10- 
26). 

With respect to the claims 19, 27, 33 and 35, it needs to be shown whether Feiner teaches 
the claim limitation of "a determination unit". Feiner teaches in Page 79 that the HTTP server 
can call a network object method that selects corresponding graphical items on the head- worn 
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display. Thus data selection works in both directions: from the backpack PC to the handheld PC 
by launching relevant URLs from the head- worn display's menus and vice versa by selecting 
buildings, departments, etc on the head- worn display from a link on the handheld' s browser. 
Feiner teaches that the tour application continuously receives input from the GPS position 
tracker and the orientation tracker and therefore, when the instruction is received from said 
instruction unit (e.g., in a graphical user interface), the mixed reality image is re-generated using 
a second calculation method and sequence such as the blending the graphics image so generated 
with a different building image and different labels are generated to overlay on the building 
image and thus requiring different calculation methods for calculating the network objects or 
generating the graphics images to be overlaid on the building images. Figs. 2-3 show the mixed 
reality images according to a first viewpoint and a second viewpoint. Based on this input and a 
database of information about campus buildings, it generates the graphics that are overlaid on the 
real world by the head-worn display. Thus, Feiner teaches a determination unit adapted to check 
a positional relationship among said image sensing unit~a position and orientation tracker with 
HMD, a real space object such as the campus building and a virtual space object such as the 
different campus building, graphics image and/or labels overlaid with the real space object on the 
coordinate system and determining whether or not the real space object is overlaid/occluded by 
the different campus building, graphics image and/or labels in the mixed reality space image. 



Claim 20: 
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Feiner teaches that a pixel size of one side of the virtual space image generated by said 
generation unit is a value within a product of a length of one side and a resolution of an image 
that said print unit can output (Feiner Section 4.1). 

Re Claims 21 and 28: 

Feiner teaches that management unit adapted to manage state information of the real and 
virtual spaces, and in that said management unit updates the state information of the virtual space 
in accordance with input information or at each predetermined timing, and said generation unit 
successively generates virtual space images in accordance with the updated state information of 
the real and virtual spaces (e.g., Section 4.2 wherein a system that provides language-level 
support for distributed virtual environment and CORTERIE is based on the distributed data- 
object paradigm for distributed shared memory and the tour application running on the 
backpack PC is response for generating the graphics and presenting it on the head-worn display 
and the HTTP server can call a network object method that selects corresponding graphical 
items on the head-worn display and the handheld display controls the head-worn display. The 
tour application continuously receives input from the GPS position tracker and the orientation 
tracker. It also takes user input from the trackpad that is physically attached to the back of the 
handheld PC. Based on this input and a database of information about campus buildings, it 
generates the graphics that are overlaid on the real world by the head-worn display). 

Re Claims 22 and 29: 

Feiner further teaches that said management unit is an independent apparatus connected 
to said image processing apparatus via a network, and the state information of the virtual space 
updated by said management unit can be received by image processing apparatuses which are 
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equivalent to said image processing apparatus, and are connected to the network (e.g., Section 
4.2 wherein a system that provides language-level support for distributed virtual environment 
and CORTERIE is based on the distributed data-object paradigm for distributed shared memory 
and the tour application running on the backpack PC is response for generating the graphics 
and presenting it on the head-worn display and the HTTP server can call a network object 
method that selects corresponding graphical items on the head-worn display and the handheld 
display controls the head-worn display. The tour application continuously receives input from 
the GPS position tracker and the orientation tracker. It also takes user input from the trackpad 
that is physically attached to the back of the handheld PC. Based on this input and a database of 
information about campus buildings, it generates the graphics that are overlaid on the real 
world by the head-worn display). 
Re Claims 23 and 30: 

Feiner further teaches that said apparatus can be simultaneously used by at least one user, 
and has said image sensing unit, said generation unit, said composition unit, said display unit, 
and detection unit adapted to detect a viewpoint position and visual axis direction of a user, 
which are prepared for each user (e.g., Section 4.2 wherein a system that provides language-level 
support for distributed virtual environment and CORTERIE is based on the distributed data- 
object paradigm for distributed shared memory and the tour application running on the 
backpack PC is response for generating the graphics and presenting it on the head-worn display 
and the HTTP server can call a network object method that selects corresponding graphical 
items on the head-worn display and the handheld display controls the head-worn display. The 
tour application continuously receives input from the GPS position tracker and the orientation 
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tracker. It also takes user input from the trackpad that is physically attached to the back of the 
handheld PC. Based on this input and a database of information about campus buildings, it 
generates the graphics that are overlaid on the real world by the head-worn display). 
Re Claims 24 and 31: 

Feiner further teaches that said display unit and said image sensing unit, forms or form a 
head-mounted display device to be mounted on a head of the user (Fig. 1). 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on (571) 272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Jin-Cheng Wang/ 

Primary Examiner, Art Unit 2628 



